{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Graph Convolution Model\n",
    "Now that we have the data appropriately formatted, we can use this data to train a Graph Convolution model.  First we need to import the necessary libraries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    }
   ],
   "source": [
    "import deepchem as dc\n",
    "from deepchem.models import GraphConvModel\n",
    "import numpy as np\n",
    "import sys\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from rdkit.Chem import PandasTools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's define a function to create a GraphConvModel.  In this case we will be creating a classification model.  Since we will be apply the model later on a different dataset, it's a good idea to create a directory in which to store the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_graph_conv_model():\n",
    "    batch_size = 128\n",
    "    model = GraphConvModel(1, batch_size=batch_size, mode='classification',model_dir=\"./model_dir\")\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will read in the dataset that we just created.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading raw samples now.\n",
      "shard_size: 8192\n",
      "About to start loading CSV from dude_erk2_mk01.csv\n",
      "Loading shard 1 of size 8192.\n",
      "Featurizing sample 0\n",
      "Featurizing sample 1000\n",
      "Featurizing sample 2000\n",
      "Featurizing sample 3000\n",
      "Featurizing sample 4000\n",
      "TIMING: featurizing shard 0 took 14.259 s\n",
      "TIMING: dataset construction took 16.347 s\n",
      "Loading dataset from disk.\n"
     ]
    }
   ],
   "source": [
    "dataset_file = \"dude_erk2_mk01.csv\"\n",
    "tasks = [\"is_active\"]\n",
    "featurizer = dc.feat.ConvMolFeaturizer()\n",
    "loader = dc.data.CSVLoader(tasks=tasks, smiles_field=\"SMILES\", featurizer=featurizer)\n",
    "dataset = loader.featurize(dataset_file, shard_size=8192)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the dataset loaded, let's build a model.\n",
    "We will create training and test sets to evaluate the model's performance. In this case we will use the RandomSplitter().  DeepChem offers a number of other splitters such as the ScaffoldSplitter, which will divide the dataset by chemical scaffold or the ButinaSplitter which will first cluster the data then split the dataset so that different clusters will end up in the training and test sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitter = dc.splits.RandomSplitter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the dataset split, we can train a model on the training set and test that model on the validation set. \n",
    "At this point we can define some metrics and evaluate the performance of our model. In this case our dataset is unbalanced, we have a small number of active compounds and a large number of inactive compounds. Given this difference, we need to use a metric that reflects the performance on unbalanced datasets. One metric that is apporpriate for datasets like this is the Matthews correlation coefficient (MCC). Put more info about MCC here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = [dc.metrics.Metric(dc.metrics.matthews_corrcoef, np.mean, mode=\"classification\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to evaluate the performance of our moldel, we will perform 10 folds of cross valiation, where we train a model on the training set and validate on the validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TIMING: dataset construction took 2.760 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.090 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.177 s\n",
      "Loading dataset from disk.\n",
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:317: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:317: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "/home/ubuntu/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "computed_metrics: [0.9733940774144629]\n",
      "computed_metrics: [0.6147334478929444]\n",
      "TIMING: dataset construction took 2.791 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.205 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.089 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.9613024323789267]\n",
      "computed_metrics: [0.7040189638958985]\n",
      "TIMING: dataset construction took 2.675 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.063 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.160 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.9560246198785713]\n",
      "computed_metrics: [0.7729109291165613]\n",
      "TIMING: dataset construction took 2.789 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.072 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.157 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.8640073855714518]\n",
      "computed_metrics: [0.6630256418076944]\n",
      "TIMING: dataset construction took 2.734 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.090 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.186 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.8874180607161601]\n",
      "computed_metrics: [0.7032364358565029]\n",
      "TIMING: dataset construction took 2.374 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.497 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.071 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.8966687876929688]\n",
      "computed_metrics: [0.8641283031407085]\n",
      "TIMING: dataset construction took 2.876 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.214 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.076 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.885610629113555]\n",
      "computed_metrics: [0.8934523381859952]\n",
      "TIMING: dataset construction took 2.759 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.189 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.300 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.8228368852646348]\n",
      "computed_metrics: [0.7534545402108427]\n",
      "TIMING: dataset construction took 2.919 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.220 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.088 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.9327612022162147]\n",
      "computed_metrics: [0.7271046235060015]\n",
      "TIMING: dataset construction took 2.716 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.198 s\n",
      "Loading dataset from disk.\n",
      "TIMING: dataset construction took 1.170 s\n",
      "Loading dataset from disk.\n",
      "computed_metrics: [0.9594842772087291]\n",
      "computed_metrics: [0.7040189638958985]\n",
      "[0.9733940774144629, 0.9613024323789267, 0.9560246198785713, 0.8640073855714518, 0.8874180607161601, 0.8966687876929688, 0.885610629113555, 0.8228368852646348, 0.9327612022162147, 0.9594842772087291]\n",
      "[0.6147334478929444, 0.7040189638958985, 0.7729109291165613, 0.6630256418076944, 0.7032364358565029, 0.8641283031407085, 0.8934523381859952, 0.7534545402108427, 0.7271046235060015, 0.7040189638958985]\n"
     ]
    }
   ],
   "source": [
    "training_score_list = []\n",
    "validation_score_list = []\n",
    "transformers = []\n",
    "cv_folds = 10\n",
    "for i in range(0,cv_folds):\n",
    "    model = generate_graph_conv_model()\n",
    "    train_dataset, valid_dataset, test_dataset = splitter.train_valid_test_split(dataset)\n",
    "    model.fit(train_dataset)\n",
    "    train_scores = model.evaluate(train_dataset, metrics, transformers)\n",
    "    training_score_list.append(train_scores[\"mean-matthews_corrcoef\"])\n",
    "    validation_scores = model.evaluate(valid_dataset, metrics, transformers)\n",
    "    validation_score_list.append(validation_scores[\"mean-matthews_corrcoef\"])\n",
    "print(training_score_list)\n",
    "print(validation_score_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the preformance of our models on the training and test data, we can make boxplots of the models' performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fce2892f390>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAVKElEQVR4nO3df4yd1Z3f8fcnY5E1P0Ng4ioDiUnHu4RGgSZTtqvsZoMSWLNSRLOKtmS3AqFuLbRgvLva1dL806YrRalQVVmGjetGFKJuQqPduDGVa0iiJLRRongMBtv8aK+MgbHTMJhoCT8a1ubbP+7j5mq443kwM77D4/dLGs19znPOvd87evhwfO6956aqkCR119tGXYAkaWkZ9JLUcQa9JHWcQS9JHWfQS1LHrRh1AcOcf/75tXr16lGXIUlvGbt27XquqsaHnVuWQb969Wqmp6dHXYYkvWUkeWq+cy7dSFLHGfSS1HEGvSR1nEEvSR1n0EtSxxn0ktRxBr0kddyyfB99F2zatIlerzfSGg4ePAjAxMTESOsAmJycZP369aMuQzolGfQd9sorr4y6BEnLgEG/RJbD7HXDhg0AbNy4ccSVSBol1+glqeMMeknqOINekjrOoJekjjPoJanjOveum+Xw/vXl4tjf4di7b051vpdfp6rOBX2v12P33sc4evo7R13KyL3t1QJg1/6fjLiS0Rt7+flRlyCNTOeCHuDo6e/klYt/e9RlaBlZ+fj2UZcgjUyrNfoka5M8kaSX5NYh589NsjXJI0l+lOQDA+cOJNmTZHcSvx9Qkk6yBWf0ScaAO4ArgRlgZ5JtVfXoQLfPArur6lNJLm76f3zg/BVV9dwi1i1JaqnNjP5yoFdV+6vqVeAe4Jo5fS4Bvg1QVY8Dq5OsWtRKJUknpE3QTwDPDBzPNG2DHgZ+ByDJ5cB7gQuacwXcn2RXknXzPUiSdUmmk0zPzs62rV+StIA2QZ8hbTXn+AvAuUl2A+uBh4AjzbmPVNWHgKuBm5J8dNiDVNWWqpqqqqnx8fF21UuSFtTmXTczwIUDxxcAhwY7VNULwA0ASQI82fxQVYea388m2Up/KeiBN125JKmVNjP6ncCaJBclOQ24Ftg22CHJO5pzAH8APFBVLyQ5I8lZTZ8zgKuAvYtXviRpIQvO6KvqSJKbgfuAMeDOqtqX5Mbm/Gbg/cCXkxwFHgX+eTN8FbC1P8lnBfCVqtqx+E9DkjSfVh+YqqrtwPY5bZsHbv8AWDNk3H7g0jdZoyTpTXBTM0nqOINekjrOoJekjjPoJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seo4g16SOs6gl6SOM+glqeMMeknquFa7V76VHDx4kLGX/5aVj29fuLNOGWMvH+bgwSMLd5Q6yBm9JHVc52b0ExMT/J+fr+CVi3971KVoGVn5+HYmJlaNugxpJJzRS1LHGfSS1HEGvSR1XKugT7I2yRNJekluHXL+3CRbkzyS5EdJPtB2rCRpaS0Y9EnGgDuAq4FLgM8kuWROt88Cu6vqg8B1wMY3MFaStITazOgvB3pVtb+qXgXuAa6Z0+cS4NsAVfU4sDrJqpZjJZ2CDh8+zC233MLhw4dHXUrntQn6CeCZgeOZpm3Qw8DvACS5HHgvcEHLsTTj1iWZTjI9OzvbrnpJb1l33303e/bs4ctf/vKoS+m8NkGfIW015/gLwLlJdgPrgYeAIy3H9hurtlTVVFVNjY+PtyhL0lvV4cOH2bFjB1XFjh07nNUvsTZBPwNcOHB8AXBosENVvVBVN1TVZfTX6MeBJ9uMlXTqufvuu3nttdcAOHr0qLP6JdYm6HcCa5JclOQ04Fpg22CHJO9ozgH8AfBAVb3QZqykU8+3vvUtjhzp7z105MgRvvnNb464om5bMOir6ghwM3Af8Bjwtaral+TGJDc23d4P7EvyOP132Gw43tjFfxqS3ko+8YlPsGJFfweWFStWcOWVV464om5rtddNVW0Hts9p2zxw+wfAmrZjJZ3arr/+enbs2AHA2NgY11133Ygr6rbObWoGMPby825TDLzt/74AwGu/dPaIKxm9sZefB9zUbLk477zzWLt2Lffeey9r167lvPPOG3VJnda5oJ+cnBx1CctGr/czACbfZ8DBKq+NZeb666/nwIEDzuZPglQNfbfjSE1NTdX09PSoy3jL27BhAwAbN24ccSWSllqSXVU1Neycm5pJUscZ9JLUcQa9JHWcQS9JHWfQS1LHGfSS1HEGvSR1nEEvSR1n0EtSx3VuCwRJx7dp0yZ6vd6oy+DgwYMATEwM/dK5k2ZycpL169ePtIalZtBLGolXXnll1CWcMgx66RSzXGav7sV08rhGL0kdZ9BLUscZ9JLUca2CPsnaJE8k6SW5dcj5c5Lcm+ThJPuS3DBw7kCSPUl2J3GTeUk6yRZ8MTbJGHAHcCUwA+xMsq2qHh3odhPwaFV9Msk48ESSv6qqV5vzV1TVc4tdvCRpYW1m9JcDvara3wT3PcA1c/oUcFaSAGcCzwNHFrVSSdIJaRP0E8AzA8czTdug24H3A4eAPcCGqnqtOVfA/Ul2JVk334MkWZdkOsn07Oxs6ycgSTq+NkGfIW1zv2j2t4DdwLuBy4Dbk5zdnPtIVX0IuBq4KclHhz1IVW2pqqmqmhofH29XvSRpQW2Cfga4cOD4Avoz90E3AF+vvh7wJHAxQFUdan4/C2ylvxQkSTpJ2gT9TmBNkouSnAZcC2yb0+dp4OMASVYBvwLsT3JGkrOa9jOAq4C9i1W8JGlhC77rpqqOJLkZuA8YA+6sqn1JbmzObwb+ArgryR76Sz1/XlXPJXkfsLX/Gi0rgK9U1Y4lei6SpCFa7XVTVduB7XPaNg/cPkR/tj533H7g0jdZoyTpTfCTsZLUcQa9JHWcQS9JHWfQS1LHGfSS1HEGvSR1nEEvSR1n0EtSx/nl4Etk06ZN9Hq9kdZw7PGPfQnzKE1OTi6bL6WWTjUGfYetXLly1CVIWgYM+iXi7FXScuEavSR1nEEvSR1n0EtSxxn0ktRxBr0kdZxBL0kdZ9BLUscZ9JLUca2CPsnaJE8k6SW5dcj5c5Lcm+ThJPuS3NB2rCRpaS0Y9EnGgDuAq4FLgM8kuWROt5uAR6vqUuBjwL9LclrLsZKkJdRmRn850Kuq/VX1KnAPcM2cPgWclSTAmcDzwJGWYyVJS6hN0E8AzwwczzRtg24H3g8cAvYAG6rqtZZjAUiyLsl0kunZ2dmW5UuSFtIm6DOkreYc/xawG3g3cBlwe5KzW47tN1ZtqaqpqpoaHx9vUZYkqY02QT8DXDhwfAH9mfugG4CvV18PeBK4uOVYSdISahP0O4E1SS5KchpwLbBtTp+ngY8DJFkF/Aqwv+VYSdISWnA/+qo6kuRm4D5gDLizqvYlubE5vxn4C+CuJHvoL9f8eVU9BzBs7NI8FUnSMK2+eKSqtgPb57RtHrh9CLiq7VhJ0snjJ2MlqeMMeknqOINekjrOoJekjjPoJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seq4VlsgSFocmzZtotfrjbqMZeHY32HDhg0jrmR5mJycZP369Uty3wa9dBL1ej3+976HeM+ZR0ddysid9nf9BYWfPzU94kpG7+kXx5b0/g166SR7z5lH+eyHXhh1GVpGPv/g2Ut6/67RS1LHGfSS1HEGvSR1nEEvSR1n0EtSx7UK+iRrkzyRpJfk1iHn/yzJ7uZnb5KjSd7ZnDuQZE9zzvdRSdJJtuDbK5OMAXcAVwIzwM4k26rq0WN9quo24Lam/yeBP66q5wfu5opjXxYuSTq52szoLwd6VbW/ql4F7gGuOU7/zwBfXYziJElvXpugnwCeGTieadpeJ8npwFrgbwaaC7g/ya4k6+Z7kCTrkkwnmZ6dnW1RliSpjTZBnyFtNU/fTwLfn7Ns85Gq+hBwNXBTko8OG1hVW6pqqqqmxsfHW5QlSWqjTdDPABcOHF8AHJqn77XMWbapqkPN72eBrfSXgiRJJ0mboN8JrElyUZLT6If5trmdkpwD/CbwjYG2M5Kcdew2cBWwdzEKlyS1s+C7bqrqSJKbgfuAMeDOqtqX5Mbm/Oam66eA+6vqpYHhq4CtSY491leqasdiPgFJ0vG12r2yqrYD2+e0bZ5zfBdw15y2/cClb6pCSdKb4idjJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seo4g16SOs6gl6SOM+glqeMMeknqOINekjrOoJekjmu1qZmkxXHw4EFe+tkYn3/w7FGXomXkqZ+NccbBg0t2/87oJanjnNFLJ9HExAQ/P/JjPvuhF0ZdipaRzz94Nm+fGPpV3IvCGb0kdZxBL0kdZ9BLUscZ9JLUca1ejE2yFthI/8vBv1RVX5hz/s+A3x+4z/cD41X1/EJjpVPN0y/69kqAn7zcn2euOv21EVcyek+/OMaaJbz/BYM+yRhwB3AlMAPsTLKtqh491qeqbgNua/p/EvjjJuQXHCudSiYnJ0ddwrLxaq8HwNvf699kDUt7bbSZ0V8O9KpqP0CSe4BrgPnC+jPAV09wrNRp69evH3UJy8aGDRsA2Lhx44gr6b42a/QTwDMDxzNN2+skOR1YC/zNCYxdl2Q6yfTs7GyLsiRJbbQJ+gxpq3n6fhL4flU9/0bHVtWWqpqqqqnx8fEWZUmS2mgT9DPAhQPHFwCH5ul7Lb9YtnmjYyVJS6BN0O8E1iS5KMlp9MN829xOSc4BfhP4xhsdK0laOgu+GFtVR5LcDNxH/y2Sd1bVviQ3Nuc3N10/BdxfVS8tNHaxn4QkaX6t3kdfVduB7XPaNs85vgu4q81YSdLJ4ydjJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seo4g16SOs6gl6SOM+glqeMMeknqOINekjrOoJekjjPoJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seq4VkGfZG2SJ5L0ktw6T5+PJdmdZF+S7w20H0iypzk3vViFS5LaWfA7Y5OMAXcAVwIzwM4k26rq0YE+7wD+ElhbVU8nedecu7miqp5bxLolSS21mdFfDvSqan9VvQrcA1wzp8/vAV+vqqcBqurZxS1TknSi2gT9BPDMwPFM0zbol4Fzk3w3ya4k1w2cK+D+pn3dfA+SZF2S6STTs7OzbeuXJC1gwaUbIEPaasj9fBj4OLAS+EGSH1bV/wI+UlWHmuWcbyZ5vKoeeN0dVm0BtgBMTU3NvX9J0glqM6OfAS4cOL4AODSkz46qeqlZi38AuBSgqg41v58FttJfCpIknSRtgn4nsCbJRUlOA64Fts3p8w3gN5KsSHI68KvAY0nOSHIWQJIzgKuAvYtXviRpIQsu3VTVkSQ3A/cBY8CdVbUvyY3N+c1V9ViSHcAjwGvAl6pqb5L3AVuTHHusr1TVjqV6MpKk12uzRk9VbQe2z2nbPOf4NuC2OW37aZZwJEmj4SdjJanjDHpJ6jiDXpI6zqCXpI4z6CWp4wx6Seo4g16SOs6gl6SOM+glqeMMeknqOINekjrOoJekjjPoJanjWu1eKak7Nm3aRK/XG3UZ/7+GDRs2jLSOyclJ1q9fP9IalppBL2kkVq5cOeoSThkGvXSK6frsVa/nGr0kdZxBL0kdZ9BLUse1Cvoka5M8kaSX5NZ5+nwsye4k+5J8742MlSQtnQVfjE0yBtwBXAnMADuTbKuqRwf6vAP4S2BtVT2d5F1tx0qSllabGf3lQK+q9lfVq8A9wDVz+vwe8PWqehqgqp59A2MlSUuoTdBPAM8MHM80bYN+GTg3yXeT7Epy3RsYC0CSdUmmk0zPzs62q16StKA276PPkLYacj8fBj4OrAR+kOSHLcf2G6u2AFsApqamhvaRJL1xbYJ+Brhw4PgC4NCQPs9V1UvAS0keAC5tOfZ1du3a9VySp1rUpoWdDzw36iKkeXh9Lp73zneiTdDvBNYkuQg4CFxLf01+0DeA25OsAE4DfhX498DjLca+TlWNt6hLLSSZrqqpUdchDeP1eXIsGPRVdSTJzcB9wBhwZ1XtS3Jjc35zVT2WZAfwCPAa8KWq2gswbOwSPRdJ0hCpcjm8y5wxaTnz+jw5/GRs920ZdQHScXh9ngTO6CWp45zRS1LHGfSS1HEG/TKU5B1J/vAExm1v9h06Xp9/k+QTJ16dNFySF5vf707y1/P0+W6S4774muSPkpw+cLzgda3jc41+GUqyGvhvVfWBOe1jVXV0JEVJC0jyYlWduUCf7wJ/WlXTx+lzAJiqKj9ItUic0S9PXwD+frPt884k30nyFWAPQJL/2uwptC/JumODkhxIcn6S1UkeS/Ifmz73J1nZ9LkryacH+n8uyYNJ9iS5uGkfT/LNpv0/JHkqyfkn/8+gUUrybwf/ZZnkXyf5V0m+PXDNvG6Twub6O/Y5mpVJ7knySJL/Qn+LlGP9vtjsb7UvyeeatluAdwPfSfKdpu3AsesvyZ8k2dv8/NHA4w293tWoKn+W2Q+wGtjb3P4Y8BJw0cD5dza/VwJ7gfOa4wP0P1K+GjgCXNa0fw34Z83tu4BPD/Rf39z+Q/ofdAO4HfiXze219PcnOn/Ufxd/Tvp1+A+B7w0cPwq8Bzi7OT4f6PGLlYEXm9+D1++f0P+gJMAHm+tyqjk+dh2PAd8FPtgcHxi83gau6w/Tn+ycAZwJ7GtqnPd696f/44z+reFHVfXkwPEtSR4Gfkh/L6E1Q8Y8WVW7m9u76P/HMMzXh/T5dfpbSlNVO4CfnnDlesuqqoeAdzVr7pfSvw5+DHw+ySPAt+jvRrvqOHfzUeA/N/f3CP1Pzx/zu0keBB4C/gFwyQIl/TqwtapeqqoX6V+7v9Gca3u9n5La7HWj0Xvp2I0kHwM+AfxaVb3crHn+0pAxPx+4fZSBfzLP0+8ov7gehu06qlPTXwOfBv4e/f/5/z4wDny4qv6uWU8fdv0Net0Lgc3+V38K/KOq+mmSu1rcz/Guy7bX+ynJGf3y9DPgrHnOnQP8tAn5i4F/vASP/z+B3wVIchVw7hI8ht4a7qG/GeGn6Yf+OcCzTchfwXF2TGw8QP9/DiT5AP3lG4Cz6U9g/jbJKuDqgTHzXf8PAP8kyelJzgA+BfyPE3pWpxhn9MtQVR1O8v3mBa1XgJ8MnN4B3Nj80/kJ+ss3i+1zwFeT/FPge/T/uf6zJXgcLXPV38DwLOBgVf04yV8B9yaZBnbT36H2eL4I/Kfmet0N/Ki534eTPER/nX0/8P2BMVuA/57kx1V1xUAtDzYz/x81TV+qqoead6npOHx7pV4nyduBo9XfufTXgC9W1WWjrkvSiXFGr2HeA3wtyduAV4F/MeJ6JL0JzuglqeN8MVaSOs6gl6SOM+glqeMMeknqOINekjru/wHVVRo9z673ggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot([\"training\"]*cv_folds+[\"validation\"]*cv_folds,training_score_list+validation_score_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also useful to visualize the result of our model.  In order to do this, we will generate a set of predictions for a validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = [x.flatten() for x in model.predict(valid_dataset)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0.78670406, 0.213296  ], dtype=float32),\n",
       " array([0.03079515, 0.9692049 ], dtype=float32),\n",
       " array([0.58493924, 0.41506073], dtype=float32),\n",
       " array([0.24829295, 0.7517071 ], dtype=float32),\n",
       " array([0.9275609, 0.0724391], dtype=float32),\n",
       " array([0.42476124, 0.57523876], dtype=float32),\n",
       " array([0.49025437, 0.5097456 ], dtype=float32),\n",
       " array([0.97195596, 0.02804398], dtype=float32),\n",
       " array([0.9975581 , 0.00244191], dtype=float32),\n",
       " array([9.999355e-01, 6.447515e-05], dtype=float32),\n",
       " array([0.99808735, 0.00191258], dtype=float32),\n",
       " array([9.9960166e-01, 3.9835321e-04], dtype=float32),\n",
       " array([9.9972945e-01, 2.7051347e-04], dtype=float32),\n",
       " array([0.99626094, 0.00373907], dtype=float32),\n",
       " array([9.9987352e-01, 1.2640837e-04], dtype=float32),\n",
       " array([0.9984927 , 0.00150731], dtype=float32),\n",
       " array([9.9959880e-01, 4.0125093e-04], dtype=float32),\n",
       " array([9.9983919e-01, 1.6080444e-04], dtype=float32),\n",
       " array([9.999192e-01, 8.087628e-05], dtype=float32),\n",
       " array([9.9987078e-01, 1.2920667e-04], dtype=float32),\n",
       " array([9.999151e-01, 8.492516e-05], dtype=float32),\n",
       " array([0.998798  , 0.00120199], dtype=float32),\n",
       " array([0.9935579 , 0.00644213], dtype=float32),\n",
       " array([9.9958426e-01, 4.1573922e-04], dtype=float32),\n",
       " array([0.9985259 , 0.00147416], dtype=float32),\n",
       " array([0.995468  , 0.00453197], dtype=float32),\n",
       " array([9.9987352e-01, 1.2640837e-04], dtype=float32),\n",
       " array([9.9988329e-01, 1.1665429e-04], dtype=float32),\n",
       " array([9.9986541e-01, 1.3456838e-04], dtype=float32),\n",
       " array([0.99498296, 0.00501696], dtype=float32),\n",
       " array([9.9958628e-01, 4.1377108e-04], dtype=float32),\n",
       " array([9.999609e-01, 3.914170e-05], dtype=float32),\n",
       " array([9.9996090e-01, 3.9141665e-05], dtype=float32),\n",
       " array([9.9970716e-01, 2.9283381e-04], dtype=float32),\n",
       " array([9.9984884e-01, 1.5118689e-04], dtype=float32),\n",
       " array([0.99344987, 0.00655011], dtype=float32),\n",
       " array([9.9995542e-01, 4.4534183e-05], dtype=float32),\n",
       " array([0.99694556, 0.00305447], dtype=float32),\n",
       " array([9.9958795e-01, 4.1201437e-04], dtype=float32),\n",
       " array([0.9894259 , 0.01057409], dtype=float32),\n",
       " array([9.9950922e-01, 4.9077783e-04], dtype=float32),\n",
       " array([0.99892753, 0.0010725 ], dtype=float32),\n",
       " array([0.9479977 , 0.05200234], dtype=float32),\n",
       " array([9.9995971e-01, 4.0283252e-05], dtype=float32),\n",
       " array([9.9945992e-01, 5.4008234e-04], dtype=float32),\n",
       " array([0.97650254, 0.02349747], dtype=float32),\n",
       " array([9.9997067e-01, 2.9369068e-05], dtype=float32),\n",
       " array([9.9957007e-01, 4.2994187e-04], dtype=float32),\n",
       " array([9.9981636e-01, 1.8371608e-04], dtype=float32),\n",
       " array([9.9994195e-01, 5.8039885e-05], dtype=float32),\n",
       " array([9.9946266e-01, 5.3739210e-04], dtype=float32),\n",
       " array([0.99863786, 0.00136218], dtype=float32),\n",
       " array([0.99396884, 0.0060312 ], dtype=float32),\n",
       " array([0.94888794, 0.05111201], dtype=float32),\n",
       " array([9.9984252e-01, 1.5749062e-04], dtype=float32),\n",
       " array([0.993654  , 0.00634598], dtype=float32),\n",
       " array([9.990495e-01, 9.505303e-04], dtype=float32),\n",
       " array([9.990651e-01, 9.349591e-04], dtype=float32),\n",
       " array([9.9994767e-01, 5.2321277e-05], dtype=float32),\n",
       " array([9.9962544e-01, 3.7458862e-04], dtype=float32),\n",
       " array([9.9987042e-01, 1.2952196e-04], dtype=float32),\n",
       " array([9.994228e-01, 5.772485e-04], dtype=float32),\n",
       " array([9.9928778e-01, 7.1220123e-04], dtype=float32),\n",
       " array([0.9968951 , 0.00310497], dtype=float32),\n",
       " array([9.9945205e-01, 5.4797815e-04], dtype=float32),\n",
       " array([9.999219e-01, 7.807566e-05], dtype=float32),\n",
       " array([9.9986911e-01, 1.3090966e-04], dtype=float32),\n",
       " array([9.9938726e-01, 6.1276834e-04], dtype=float32),\n",
       " array([9.9951696e-01, 4.8303549e-04], dtype=float32),\n",
       " array([9.9929655e-01, 7.0347451e-04], dtype=float32),\n",
       " array([9.9985182e-01, 1.4811756e-04], dtype=float32),\n",
       " array([9.9979573e-01, 2.0431611e-04], dtype=float32),\n",
       " array([0.9988024 , 0.00119761], dtype=float32),\n",
       " array([0.9985582 , 0.00144177], dtype=float32),\n",
       " array([9.9989009e-01, 1.0990025e-04], dtype=float32),\n",
       " array([0.9987484 , 0.00125156], dtype=float32),\n",
       " array([9.9943191e-01, 5.6801335e-04], dtype=float32),\n",
       " array([0.9913044 , 0.00869565], dtype=float32),\n",
       " array([0.9977677 , 0.00223232], dtype=float32),\n",
       " array([0.9675066 , 0.03249339], dtype=float32),\n",
       " array([0.99893135, 0.0010687 ], dtype=float32),\n",
       " array([9.9965703e-01, 3.4299758e-04], dtype=float32),\n",
       " array([0.99842227, 0.00157779], dtype=float32),\n",
       " array([9.9987233e-01, 1.2768575e-04], dtype=float32),\n",
       " array([9.9936134e-01, 6.3866435e-04], dtype=float32),\n",
       " array([9.9980861e-01, 1.9142128e-04], dtype=float32),\n",
       " array([0.9964126 , 0.00358743], dtype=float32),\n",
       " array([0.7354119 , 0.26458812], dtype=float32),\n",
       " array([0.9907637 , 0.00923627], dtype=float32),\n",
       " array([9.9981540e-01, 1.8457553e-04], dtype=float32),\n",
       " array([0.9974801 , 0.00251996], dtype=float32),\n",
       " array([9.9910331e-01, 8.9672825e-04], dtype=float32),\n",
       " array([9.9927133e-01, 7.2866469e-04], dtype=float32),\n",
       " array([9.9951673e-01, 4.8324803e-04], dtype=float32),\n",
       " array([9.9952734e-01, 4.7271745e-04], dtype=float32),\n",
       " array([0.9932874 , 0.00671259], dtype=float32),\n",
       " array([9.9994278e-01, 5.7188518e-05], dtype=float32),\n",
       " array([0.9977576 , 0.00224244], dtype=float32),\n",
       " array([9.9921238e-01, 7.8763487e-04], dtype=float32),\n",
       " array([0.99745816, 0.00254185], dtype=float32),\n",
       " array([0.99745387, 0.00254614], dtype=float32),\n",
       " array([0.9989178 , 0.00108217], dtype=float32),\n",
       " array([0.99824   , 0.00175996], dtype=float32),\n",
       " array([9.998192e-01, 1.808385e-04], dtype=float32),\n",
       " array([9.992299e-01, 7.701112e-04], dtype=float32),\n",
       " array([9.9960953e-01, 3.9051921e-04], dtype=float32),\n",
       " array([9.9929285e-01, 7.0712384e-04], dtype=float32),\n",
       " array([0.9908199 , 0.00918009], dtype=float32),\n",
       " array([0.96046776, 0.03953223], dtype=float32),\n",
       " array([9.9944586e-01, 5.5409846e-04], dtype=float32),\n",
       " array([9.9923754e-01, 7.6247047e-04], dtype=float32),\n",
       " array([0.9923498 , 0.00765011], dtype=float32),\n",
       " array([9.9982125e-01, 1.7884014e-04], dtype=float32),\n",
       " array([9.9964714e-01, 3.5285059e-04], dtype=float32),\n",
       " array([9.997235e-01, 2.764648e-04], dtype=float32),\n",
       " array([9.9919933e-01, 8.0071785e-04], dtype=float32),\n",
       " array([0.9984127 , 0.00158726], dtype=float32),\n",
       " array([9.9954599e-01, 4.5392587e-04], dtype=float32),\n",
       " array([0.9953009 , 0.00469907], dtype=float32),\n",
       " array([9.9971229e-01, 2.8764363e-04], dtype=float32),\n",
       " array([0.9985561 , 0.00144386], dtype=float32),\n",
       " array([9.991986e-01, 8.014541e-04], dtype=float32),\n",
       " array([0.99369687, 0.00630319], dtype=float32),\n",
       " array([9.995202e-01, 4.797925e-04], dtype=float32),\n",
       " array([0.9987594, 0.0012406], dtype=float32),\n",
       " array([9.9954116e-01, 4.5887069e-04], dtype=float32),\n",
       " array([0.99866843, 0.00133162], dtype=float32),\n",
       " array([9.9984467e-01, 1.5526591e-04], dtype=float32),\n",
       " array([9.9977010e-01, 2.2990005e-04], dtype=float32),\n",
       " array([9.992011e-01, 7.989565e-04], dtype=float32),\n",
       " array([0.99806887, 0.0019312 ], dtype=float32),\n",
       " array([0.9882776 , 0.01172245], dtype=float32),\n",
       " array([0.9917995, 0.0082005], dtype=float32),\n",
       " array([0.99663526, 0.0033648 ], dtype=float32),\n",
       " array([9.9973422e-01, 2.6576946e-04], dtype=float32),\n",
       " array([9.9918491e-01, 8.1511086e-04], dtype=float32),\n",
       " array([9.9984217e-01, 1.5784723e-04], dtype=float32),\n",
       " array([0.9798565 , 0.02014352], dtype=float32),\n",
       " array([9.9949765e-01, 5.0228788e-04], dtype=float32),\n",
       " array([0.99232554, 0.00767443], dtype=float32),\n",
       " array([9.9988484e-01, 1.1517804e-04], dtype=float32),\n",
       " array([9.9915195e-01, 8.4806583e-04], dtype=float32),\n",
       " array([0.996878  , 0.00312202], dtype=float32),\n",
       " array([0.9984334 , 0.00156659], dtype=float32),\n",
       " array([9.99896765e-01, 1.03230006e-04], dtype=float32),\n",
       " array([9.999260e-01, 7.404432e-05], dtype=float32),\n",
       " array([0.99884486, 0.00115508], dtype=float32),\n",
       " array([9.9923551e-01, 7.6447666e-04], dtype=float32),\n",
       " array([9.9913919e-01, 8.6076243e-04], dtype=float32),\n",
       " array([9.9957496e-01, 4.2508027e-04], dtype=float32),\n",
       " array([9.9960250e-01, 3.9748527e-04], dtype=float32),\n",
       " array([9.9962640e-01, 3.7359164e-04], dtype=float32),\n",
       " array([0.99640816, 0.00359178], dtype=float32),\n",
       " array([9.9978369e-01, 2.1631479e-04], dtype=float32),\n",
       " array([9.9942786e-01, 5.7205727e-04], dtype=float32),\n",
       " array([9.9928588e-01, 7.1420363e-04], dtype=float32),\n",
       " array([9.998994e-01, 1.006347e-04], dtype=float32),\n",
       " array([0.9955836 , 0.00441645], dtype=float32),\n",
       " array([0.9963509 , 0.00364909], dtype=float32),\n",
       " array([9.9964404e-01, 3.5596234e-04], dtype=float32),\n",
       " array([9.9972504e-01, 2.7496830e-04], dtype=float32),\n",
       " array([0.9546059 , 0.04539407], dtype=float32),\n",
       " array([0.98907787, 0.0109221 ], dtype=float32),\n",
       " array([0.9980988 , 0.00190121], dtype=float32),\n",
       " array([9.993062e-01, 6.937816e-04], dtype=float32),\n",
       " array([0.9978154 , 0.00218466], dtype=float32),\n",
       " array([0.9955277 , 0.00447227], dtype=float32),\n",
       " array([9.9986029e-01, 1.3972758e-04], dtype=float32),\n",
       " array([0.97613996, 0.02386   ], dtype=float32),\n",
       " array([0.998211  , 0.00178898], dtype=float32),\n",
       " array([0.9968148 , 0.00318523], dtype=float32),\n",
       " array([0.9989003 , 0.00109965], dtype=float32),\n",
       " array([0.99704045, 0.0029595 ], dtype=float32),\n",
       " array([9.9923575e-01, 7.6421583e-04], dtype=float32),\n",
       " array([9.9974209e-01, 2.5787213e-04], dtype=float32),\n",
       " array([9.993741e-01, 6.259666e-04], dtype=float32),\n",
       " array([9.9957043e-01, 4.2960656e-04], dtype=float32),\n",
       " array([9.9954200e-01, 4.5808844e-04], dtype=float32),\n",
       " array([0.9775572 , 0.02244275], dtype=float32),\n",
       " array([0.9954437 , 0.00455636], dtype=float32),\n",
       " array([0.9910131 , 0.00898689], dtype=float32),\n",
       " array([0.89200854, 0.10799146], dtype=float32),\n",
       " array([0.9988927 , 0.00110728], dtype=float32),\n",
       " array([9.9925631e-01, 7.4363267e-04], dtype=float32),\n",
       " array([9.9986613e-01, 1.3385901e-04], dtype=float32),\n",
       " array([0.99760205, 0.00239794], dtype=float32),\n",
       " array([0.9951357 , 0.00486426], dtype=float32),\n",
       " array([9.990270e-01, 9.730389e-04], dtype=float32),\n",
       " array([9.995049e-01, 4.951113e-04], dtype=float32),\n",
       " array([9.997341e-01, 2.659251e-04], dtype=float32),\n",
       " array([0.998531  , 0.00146899], dtype=float32),\n",
       " array([9.9900633e-01, 9.9366193e-04], dtype=float32),\n",
       " array([9.9956125e-01, 4.3878029e-04], dtype=float32),\n",
       " array([9.999080e-01, 9.208034e-05], dtype=float32),\n",
       " array([0.99883753, 0.00116248], dtype=float32),\n",
       " array([0.99771214, 0.0022879 ], dtype=float32),\n",
       " array([9.9926013e-01, 7.3988235e-04], dtype=float32),\n",
       " array([9.9987102e-01, 1.2902569e-04], dtype=float32),\n",
       " array([0.92656   , 0.07344002], dtype=float32),\n",
       " array([0.99896336, 0.00103667], dtype=float32),\n",
       " array([9.9954396e-01, 4.5603284e-04], dtype=float32),\n",
       " array([9.9929416e-01, 7.0585648e-04], dtype=float32),\n",
       " array([9.9956900e-01, 4.3102668e-04], dtype=float32),\n",
       " array([9.9930000e-01, 6.9999124e-04], dtype=float32),\n",
       " array([0.99833786, 0.00166207], dtype=float32),\n",
       " array([0.9967204 , 0.00327966], dtype=float32),\n",
       " array([9.993856e-01, 6.144029e-04], dtype=float32),\n",
       " array([0.9974516 , 0.00254841], dtype=float32),\n",
       " array([9.9981493e-01, 1.8504144e-04], dtype=float32),\n",
       " array([9.9982893e-01, 1.7098279e-04], dtype=float32),\n",
       " array([0.98622966, 0.01377041], dtype=float32),\n",
       " array([0.998792, 0.001208], dtype=float32),\n",
       " array([9.9980396e-01, 1.9607737e-04], dtype=float32),\n",
       " array([9.9994338e-01, 5.6569417e-05], dtype=float32),\n",
       " array([0.9979639 , 0.00203607], dtype=float32),\n",
       " array([0.9989881 , 0.00101192], dtype=float32),\n",
       " array([9.994147e-01, 5.853257e-04], dtype=float32),\n",
       " array([0.99811304, 0.001887  ], dtype=float32),\n",
       " array([9.9985063e-01, 1.4940242e-04], dtype=float32),\n",
       " array([9.9943715e-01, 5.6285795e-04], dtype=float32),\n",
       " array([0.9978865 , 0.00211356], dtype=float32),\n",
       " array([9.997197e-01, 2.803585e-04], dtype=float32),\n",
       " array([9.990903e-01, 9.096434e-04], dtype=float32),\n",
       " array([9.9982423e-01, 1.7581112e-04], dtype=float32),\n",
       " array([0.988362  , 0.01163798], dtype=float32),\n",
       " array([9.9979264e-01, 2.0735692e-04], dtype=float32),\n",
       " array([9.9992967e-01, 7.0358125e-05], dtype=float32),\n",
       " array([9.9989879e-01, 1.0119852e-04], dtype=float32),\n",
       " array([9.9986064e-01, 1.3933587e-04], dtype=float32),\n",
       " array([0.99142975, 0.00857021], dtype=float32),\n",
       " array([9.999087e-01, 9.131436e-05], dtype=float32),\n",
       " array([0.9925471 , 0.00745286], dtype=float32),\n",
       " array([9.995435e-01, 4.564838e-04], dtype=float32),\n",
       " array([9.9939823e-01, 6.0179067e-04], dtype=float32),\n",
       " array([9.9972218e-01, 2.7784627e-04], dtype=float32),\n",
       " array([0.9871975 , 0.01280252], dtype=float32),\n",
       " array([9.9995852e-01, 4.1484276e-05], dtype=float32),\n",
       " array([0.9727728 , 0.02722716], dtype=float32),\n",
       " array([9.9985933e-01, 1.4070132e-04], dtype=float32),\n",
       " array([9.999418e-01, 5.819368e-05], dtype=float32),\n",
       " array([9.999337e-01, 6.625894e-05], dtype=float32),\n",
       " array([9.9970263e-01, 2.9730579e-04], dtype=float32),\n",
       " array([9.9942952e-01, 5.7044695e-04], dtype=float32),\n",
       " array([0.99399734, 0.00600266], dtype=float32),\n",
       " array([9.9990654e-01, 9.3416515e-05], dtype=float32),\n",
       " array([9.99887586e-01, 1.12348236e-04], dtype=float32),\n",
       " array([0.9960991, 0.0039009], dtype=float32),\n",
       " array([0.99895763, 0.0010423 ], dtype=float32),\n",
       " array([0.99232894, 0.00767112], dtype=float32),\n",
       " array([9.9967837e-01, 3.2162233e-04], dtype=float32),\n",
       " array([0.99613154, 0.00386851], dtype=float32),\n",
       " array([9.996536e-01, 3.464494e-04], dtype=float32),\n",
       " array([0.99878615, 0.00121386], dtype=float32),\n",
       " array([9.9988258e-01, 1.1736817e-04], dtype=float32),\n",
       " array([0.89637077, 0.10362922], dtype=float32),\n",
       " array([9.9980408e-01, 1.9591142e-04], dtype=float32),\n",
       " array([0.996494  , 0.00350602], dtype=float32),\n",
       " array([9.9988914e-01, 1.1083437e-04], dtype=float32),\n",
       " array([0.9987953 , 0.00120467], dtype=float32),\n",
       " array([9.9968278e-01, 3.1724464e-04], dtype=float32),\n",
       " array([9.9937421e-01, 6.2584726e-04], dtype=float32),\n",
       " array([0.98428863, 0.01571138], dtype=float32),\n",
       " array([0.91845727, 0.08154279], dtype=float32),\n",
       " array([9.9993765e-01, 6.2307241e-05], dtype=float32),\n",
       " array([9.9961519e-01, 3.8482458e-04], dtype=float32),\n",
       " array([0.64371085, 0.35628915], dtype=float32),\n",
       " array([9.992487e-01, 7.513450e-04], dtype=float32),\n",
       " array([0.9916928, 0.0083073], dtype=float32),\n",
       " array([9.9980015e-01, 1.9981846e-04], dtype=float32),\n",
       " array([0.94978213, 0.05021779], dtype=float32),\n",
       " array([0.99603075, 0.00396929], dtype=float32),\n",
       " array([0.9131817, 0.0868183], dtype=float32),\n",
       " array([0.95582336, 0.04417666], dtype=float32),\n",
       " array([0.99722123, 0.00277878], dtype=float32),\n",
       " array([0.997698  , 0.00230201], dtype=float32),\n",
       " array([0.99898237, 0.00101766], dtype=float32),\n",
       " array([9.9968648e-01, 3.1356144e-04], dtype=float32),\n",
       " array([9.9974984e-01, 2.5018962e-04], dtype=float32),\n",
       " array([9.9977976e-01, 2.2019174e-04], dtype=float32),\n",
       " array([9.9956292e-01, 4.3701555e-04], dtype=float32),\n",
       " array([0.9956573 , 0.00434271], dtype=float32),\n",
       " array([0.9869903 , 0.01300973], dtype=float32),\n",
       " array([9.990281e-01, 9.719136e-04], dtype=float32),\n",
       " array([9.999089e-01, 9.101295e-05], dtype=float32),\n",
       " array([9.992705e-01, 7.295405e-04], dtype=float32),\n",
       " array([0.99351984, 0.00648013], dtype=float32),\n",
       " array([0.98884284, 0.01115715], dtype=float32),\n",
       " array([9.9986768e-01, 1.3228621e-04], dtype=float32),\n",
       " array([9.9953616e-01, 4.6385918e-04], dtype=float32),\n",
       " array([9.9958879e-01, 4.1118352e-04], dtype=float32),\n",
       " array([0.99675083, 0.00324923], dtype=float32),\n",
       " array([0.9948744 , 0.00512556], dtype=float32),\n",
       " array([9.9986327e-01, 1.3666783e-04], dtype=float32),\n",
       " array([9.9922264e-01, 7.7732414e-04], dtype=float32),\n",
       " array([9.9982733e-01, 1.7266175e-04], dtype=float32),\n",
       " array([9.9967957e-01, 3.2042075e-04], dtype=float32),\n",
       " array([9.9986815e-01, 1.3180322e-04], dtype=float32),\n",
       " array([9.9979073e-01, 2.0931273e-04], dtype=float32),\n",
       " array([9.9959666e-01, 4.0341428e-04], dtype=float32),\n",
       " array([9.9968123e-01, 3.1882952e-04], dtype=float32),\n",
       " array([0.9986407 , 0.00135934], dtype=float32),\n",
       " array([0.9984303 , 0.00156968], dtype=float32),\n",
       " array([9.9945182e-01, 5.4812746e-04], dtype=float32),\n",
       " array([9.9987793e-01, 1.2199949e-04], dtype=float32),\n",
       " array([9.9929130e-01, 7.0861704e-04], dtype=float32),\n",
       " array([9.9948347e-01, 5.1645021e-04], dtype=float32),\n",
       " array([0.9982551 , 0.00174492], dtype=float32),\n",
       " array([9.9979514e-01, 2.0488145e-04], dtype=float32),\n",
       " array([9.9987006e-01, 1.2991867e-04], dtype=float32),\n",
       " array([9.9957377e-01, 4.2623814e-04], dtype=float32),\n",
       " array([0.9963965 , 0.00360353], dtype=float32),\n",
       " array([0.9987041 , 0.00129591], dtype=float32),\n",
       " array([9.9973565e-01, 2.6433586e-04], dtype=float32),\n",
       " array([9.9970633e-01, 2.9362045e-04], dtype=float32),\n",
       " array([9.9963999e-01, 3.6004253e-04], dtype=float32),\n",
       " array([0.9974837 , 0.00251632], dtype=float32),\n",
       " array([9.996124e-01, 3.876064e-04], dtype=float32),\n",
       " array([9.9991035e-01, 8.9585737e-05], dtype=float32),\n",
       " array([9.9994862e-01, 5.1356732e-05], dtype=float32),\n",
       " array([9.9991155e-01, 8.8406123e-05], dtype=float32),\n",
       " array([9.9957269e-01, 4.2726734e-04], dtype=float32),\n",
       " array([9.9990773e-01, 9.2300907e-05], dtype=float32),\n",
       " array([0.99528426, 0.00471573], dtype=float32),\n",
       " array([9.9972814e-01, 2.7180556e-04], dtype=float32),\n",
       " array([0.9968888 , 0.00311124], dtype=float32),\n",
       " array([9.995480e-01, 4.519273e-04], dtype=float32),\n",
       " array([0.9959066, 0.0040934], dtype=float32),\n",
       " array([9.9981850e-01, 1.8151448e-04], dtype=float32),\n",
       " array([9.9973625e-01, 2.6370454e-04], dtype=float32),\n",
       " array([9.9935597e-01, 6.4401265e-04], dtype=float32),\n",
       " array([0.99874157, 0.0012585 ], dtype=float32),\n",
       " array([9.9986553e-01, 1.3447912e-04], dtype=float32),\n",
       " array([0.97740287, 0.0225971 ], dtype=float32),\n",
       " array([9.9983752e-01, 1.6243517e-04], dtype=float32),\n",
       " array([9.9988556e-01, 1.1443644e-04], dtype=float32),\n",
       " array([9.994708e-01, 5.292219e-04], dtype=float32),\n",
       " array([9.9988353e-01, 1.1643459e-04], dtype=float32),\n",
       " array([9.9989116e-01, 1.0882061e-04], dtype=float32),\n",
       " array([9.9991405e-01, 8.5912696e-05], dtype=float32),\n",
       " array([0.99881727, 0.00118274], dtype=float32),\n",
       " array([9.997242e-01, 2.757819e-04], dtype=float32),\n",
       " array([0.99734914, 0.0026509 ], dtype=float32),\n",
       " array([0.9982541 , 0.00174591], dtype=float32),\n",
       " array([9.9960786e-01, 3.9210721e-04], dtype=float32),\n",
       " array([0.98589754, 0.0141024 ], dtype=float32),\n",
       " array([0.99618995, 0.00381005], dtype=float32),\n",
       " array([0.998166  , 0.00183406], dtype=float32),\n",
       " array([9.9982423e-01, 1.7582253e-04], dtype=float32),\n",
       " array([9.9976474e-01, 2.3527621e-04], dtype=float32),\n",
       " array([0.9985031 , 0.00149692], dtype=float32),\n",
       " array([0.9984546 , 0.00154544], dtype=float32),\n",
       " array([9.9953270e-01, 4.6737352e-04], dtype=float32),\n",
       " array([9.9987638e-01, 1.2355752e-04], dtype=float32),\n",
       " array([0.9989089 , 0.00109113], dtype=float32),\n",
       " array([9.994382e-01, 5.617785e-04], dtype=float32),\n",
       " array([0.9985556 , 0.00144445], dtype=float32),\n",
       " array([9.9990857e-01, 9.1420821e-05], dtype=float32),\n",
       " array([0.99803907, 0.00196093], dtype=float32),\n",
       " array([0.998367  , 0.00163297], dtype=float32),\n",
       " array([9.9935478e-01, 6.4520474e-04], dtype=float32),\n",
       " array([9.9968684e-01, 3.1319665e-04], dtype=float32),\n",
       " array([0.9976987 , 0.00230128], dtype=float32),\n",
       " array([9.9952006e-01, 4.8000022e-04], dtype=float32),\n",
       " array([9.9995542e-01, 4.4531465e-05], dtype=float32),\n",
       " array([0.98856354, 0.01143642], dtype=float32),\n",
       " array([9.9989200e-01, 1.0799847e-04], dtype=float32),\n",
       " array([0.99808633, 0.00191369], dtype=float32),\n",
       " array([0.9952217 , 0.00477833], dtype=float32),\n",
       " array([0.99890697, 0.00109302], dtype=float32),\n",
       " array([9.9966335e-01, 3.3664520e-04], dtype=float32),\n",
       " array([9.9934882e-01, 6.5113197e-04], dtype=float32),\n",
       " array([9.9983382e-01, 1.6617928e-04], dtype=float32),\n",
       " array([9.993042e-01, 6.958675e-04], dtype=float32),\n",
       " array([0.9984321 , 0.00156785], dtype=float32),\n",
       " array([9.9969244e-01, 3.0756913e-04], dtype=float32),\n",
       " array([0.997682  , 0.00231803], dtype=float32),\n",
       " array([0.989242  , 0.01075805], dtype=float32),\n",
       " array([9.9938273e-01, 6.1727921e-04], dtype=float32),\n",
       " array([9.9955875e-01, 4.4123208e-04], dtype=float32),\n",
       " array([0.9606185 , 0.03938142], dtype=float32),\n",
       " array([9.9928051e-01, 7.1952306e-04], dtype=float32),\n",
       " array([0.9807015 , 0.01929846], dtype=float32),\n",
       " array([9.996438e-01, 3.561409e-04], dtype=float32),\n",
       " array([0.99689865, 0.00310134], dtype=float32),\n",
       " array([0.9984205 , 0.00157957], dtype=float32),\n",
       " array([0.9974921 , 0.00250791], dtype=float32),\n",
       " array([9.9994373e-01, 5.6317735e-05], dtype=float32),\n",
       " array([9.9980253e-01, 1.9753336e-04], dtype=float32),\n",
       " array([9.9965739e-01, 3.4264068e-04], dtype=float32),\n",
       " array([0.99790454, 0.00209543], dtype=float32),\n",
       " array([0.99808717, 0.00191282], dtype=float32),\n",
       " array([0.99853003, 0.00146994], dtype=float32),\n",
       " array([9.99102e-01, 8.97974e-04], dtype=float32),\n",
       " array([0.99819   , 0.00181004], dtype=float32),\n",
       " array([9.9989462e-01, 1.0538745e-04], dtype=float32),\n",
       " array([0.9920948 , 0.00790521], dtype=float32),\n",
       " array([9.995951e-01, 4.048545e-04], dtype=float32),\n",
       " array([0.97094   , 0.02906001], dtype=float32),\n",
       " array([9.9969959e-01, 3.0045077e-04], dtype=float32),\n",
       " array([9.999304e-01, 6.966286e-05], dtype=float32),\n",
       " array([9.9986017e-01, 1.3976554e-04], dtype=float32),\n",
       " array([9.9972337e-01, 2.7659108e-04], dtype=float32),\n",
       " array([0.9973508 , 0.00264923], dtype=float32),\n",
       " array([9.9915278e-01, 8.4717775e-04], dtype=float32),\n",
       " array([0.9987154 , 0.00128462], dtype=float32),\n",
       " array([9.9970406e-01, 2.9586852e-04], dtype=float32),\n",
       " array([9.9965799e-01, 3.4197507e-04], dtype=float32),\n",
       " array([0.998216  , 0.00178399], dtype=float32),\n",
       " array([0.9976107 , 0.00238934], dtype=float32),\n",
       " array([9.9904531e-01, 9.5463893e-04], dtype=float32),\n",
       " array([0.9980045 , 0.00199556], dtype=float32),\n",
       " array([9.998685e-01, 1.314588e-04], dtype=float32),\n",
       " array([0.97977096, 0.02022907], dtype=float32),\n",
       " array([9.994462e-01, 5.538297e-04], dtype=float32),\n",
       " array([9.9988711e-01, 1.1286032e-04], dtype=float32),\n",
       " array([9.9989855e-01, 1.0139693e-04], dtype=float32),\n",
       " array([9.9988735e-01, 1.1268031e-04], dtype=float32),\n",
       " array([9.9961323e-01, 3.8682768e-04], dtype=float32),\n",
       " array([0.99829787, 0.0017021 ], dtype=float32),\n",
       " array([0.9515473 , 0.04845264], dtype=float32),\n",
       " array([9.9991870e-01, 8.1276215e-05], dtype=float32),\n",
       " array([9.9992347e-01, 7.6545730e-05], dtype=float32),\n",
       " array([9.9984229e-01, 1.5765047e-04], dtype=float32),\n",
       " array([9.9906224e-01, 9.3776215e-04], dtype=float32),\n",
       " array([9.9961406e-01, 3.8595949e-04], dtype=float32),\n",
       " array([9.99887705e-01, 1.12277026e-04], dtype=float32),\n",
       " array([9.9970156e-01, 2.9835326e-04], dtype=float32),\n",
       " array([9.9965835e-01, 3.4165379e-04], dtype=float32),\n",
       " array([9.9945086e-01, 5.4911611e-04], dtype=float32),\n",
       " array([9.9988413e-01, 1.1589348e-04], dtype=float32),\n",
       " array([9.9905926e-01, 9.4069552e-04], dtype=float32),\n",
       " array([9.9959785e-01, 4.0222408e-04], dtype=float32),\n",
       " array([9.9972111e-01, 2.7881746e-04], dtype=float32),\n",
       " array([9.9967194e-01, 3.2810128e-04], dtype=float32),\n",
       " array([0.9982066 , 0.00179341], dtype=float32),\n",
       " array([9.9993944e-01, 6.0559050e-05], dtype=float32),\n",
       " array([0.9973369 , 0.00266311], dtype=float32),\n",
       " array([0.9987581, 0.0012419], dtype=float32),\n",
       " array([0.99791664, 0.00208337], dtype=float32),\n",
       " array([0.99880457, 0.00119539], dtype=float32),\n",
       " array([9.9972314e-01, 2.7686360e-04], dtype=float32),\n",
       " array([9.9978954e-01, 2.1042148e-04], dtype=float32),\n",
       " array([9.9990737e-01, 9.2575654e-05], dtype=float32),\n",
       " array([0.99886346, 0.00113658], dtype=float32),\n",
       " array([0.9988819 , 0.00111815], dtype=float32),\n",
       " array([9.9986541e-01, 1.3458123e-04], dtype=float32),\n",
       " array([9.990977e-01, 9.022691e-04], dtype=float32),\n",
       " array([9.9979311e-01, 2.0685808e-04], dtype=float32),\n",
       " array([9.9975032e-01, 2.4965324e-04], dtype=float32),\n",
       " array([9.9961531e-01, 3.8466539e-04], dtype=float32),\n",
       " array([0.99847263, 0.00152731], dtype=float32),\n",
       " array([9.9977833e-01, 2.2162948e-04], dtype=float32),\n",
       " array([9.9947089e-01, 5.2916544e-04], dtype=float32),\n",
       " array([0.94995856, 0.05004145], dtype=float32),\n",
       " array([9.992331e-01, 7.668626e-04], dtype=float32),\n",
       " array([9.99897718e-01, 1.02274105e-04], dtype=float32),\n",
       " array([9.9985611e-01, 1.4390376e-04], dtype=float32),\n",
       " array([9.9978715e-01, 2.1285811e-04], dtype=float32),\n",
       " array([9.9973148e-01, 2.6850204e-04], dtype=float32),\n",
       " array([9.9979895e-01, 2.0111902e-04], dtype=float32),\n",
       " array([0.9965861 , 0.00341392], dtype=float32),\n",
       " array([0.97046024, 0.02953972], dtype=float32),\n",
       " array([0.99598   , 0.00401991], dtype=float32)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The results of predict on a GraphConv model are returned as a list of lists.  Is this the intent? It doesn't seem consistent across models.  RandomForest returns a list. For convenience, we will put our predicted results into a Pandas dataframe.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_df = pd.DataFrame(pred,columns=[\"neg\",\"pos\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily add the activity class (1 = active, 0 = inactive) and the SMILES string for our predicted moleculesto the dataframe.  __Is the moleculed id retained as part of the DeepChem dataset? I can't find it__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_df[\"active\"] = [int(x) for x in valid_dataset.y]\n",
    "pred_df[\"SMILES\"] = valid_dataset.ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg</th>\n",
       "      <th>pos</th>\n",
       "      <th>active</th>\n",
       "      <th>SMILES</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.786704</td>\n",
       "      <td>0.213296</td>\n",
       "      <td>1</td>\n",
       "      <td>Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCC...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.030795</td>\n",
       "      <td>0.969205</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1cnc(nc1c2cc([nH]c2)C(=O)N[C@H](CO)c3cccc(c3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.584939</td>\n",
       "      <td>0.415061</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.248293</td>\n",
       "      <td>0.751707</td>\n",
       "      <td>1</td>\n",
       "      <td>Cn1c-2c(c(n1)C(=O)N)CCc3c2nc(nc3)NC4CCN(CC4)C(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.927561</td>\n",
       "      <td>0.072439</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        neg       pos  active  \\\n",
       "0  0.786704  0.213296       1   \n",
       "1  0.030795  0.969205       1   \n",
       "2  0.584939  0.415061       1   \n",
       "3  0.248293  0.751707       1   \n",
       "4  0.927561  0.072439       1   \n",
       "\n",
       "                                              SMILES  \n",
       "0  Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCC...  \n",
       "1  Cc1cnc(nc1c2cc([nH]c2)C(=O)N[C@H](CO)c3cccc(c3...  \n",
       "2      Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl  \n",
       "3  Cn1c-2c(c(n1)C(=O)N)CCc3c2nc(nc3)NC4CCN(CC4)C(...  \n",
       "4   Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg</th>\n",
       "      <th>pos</th>\n",
       "      <th>active</th>\n",
       "      <th>SMILES</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.030795</td>\n",
       "      <td>0.969205</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1cnc(nc1c2cc([nH]c2)C(=O)N[C@H](CO)c3cccc(c3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.248293</td>\n",
       "      <td>0.751707</td>\n",
       "      <td>1</td>\n",
       "      <td>Cn1c-2c(c(n1)C(=O)N)CCc3c2nc(nc3)NC4CCN(CC4)C(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.424761</td>\n",
       "      <td>0.575239</td>\n",
       "      <td>1</td>\n",
       "      <td>CNC(=O)Nc1ccc(cn1)CNc2c(scn2)C(=O)Nc3ccc4c(c3)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.490254</td>\n",
       "      <td>0.509746</td>\n",
       "      <td>1</td>\n",
       "      <td>CCN(C)C(=O)c1cc(c[nH]1)c2c(cn[nH]2)c3cccc(c3)Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.584939</td>\n",
       "      <td>0.415061</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>0.643711</td>\n",
       "      <td>0.356289</td>\n",
       "      <td>0</td>\n",
       "      <td>CCN[C@H](c1cccs1)c1cc2[nH]c(=O)[nH]c2cc1Br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.735412</td>\n",
       "      <td>0.264588</td>\n",
       "      <td>0</td>\n",
       "      <td>Cc1c2c([nH]c1C(=O)NCCN3c4ccccc4N[C@@H]3C)CCCC2=O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.786704</td>\n",
       "      <td>0.213296</td>\n",
       "      <td>1</td>\n",
       "      <td>Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCC...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>0.892009</td>\n",
       "      <td>0.107991</td>\n",
       "      <td>0</td>\n",
       "      <td>c1ccc(cc1)c2nc([nH]n2)CN3c4ccccc4N[C@@H]3C5CC5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>0.896371</td>\n",
       "      <td>0.103629</td>\n",
       "      <td>0</td>\n",
       "      <td>c1c(c(cc(c1N)F)F)C(=O)N(CCO)CCO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>0.913182</td>\n",
       "      <td>0.086818</td>\n",
       "      <td>0</td>\n",
       "      <td>c1cc(oc1)C(=O)N(c2nc3ccc(cc3s2)F)/N=C/c4cccs4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>0.918457</td>\n",
       "      <td>0.081543</td>\n",
       "      <td>0</td>\n",
       "      <td>N=C(N)c1cccc(OCCCOc2cccc(C(F)(F)F)c2)c1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>0.926560</td>\n",
       "      <td>0.073440</td>\n",
       "      <td>0</td>\n",
       "      <td>N=c1[nH]c(N2CCN3C[C@@H](CNCc4cccnc4)CC[C@H]3C2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.927561</td>\n",
       "      <td>0.072439</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>0.947998</td>\n",
       "      <td>0.052002</td>\n",
       "      <td>0</td>\n",
       "      <td>c1c(cc(cc1Cl)Cl)n2c3c(cn2)c(ncn3)NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>0.948888</td>\n",
       "      <td>0.051112</td>\n",
       "      <td>0</td>\n",
       "      <td>c1ccc2c(c1)nc(s2)Nc3c(c(ncn3)Nc4cccc5c4nccc5)N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>0.949782</td>\n",
       "      <td>0.050218</td>\n",
       "      <td>0</td>\n",
       "      <td>Cc1ccccc1c2nc(on2)CN3CCN(CC3)c4ccccc4C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453</th>\n",
       "      <td>0.949959</td>\n",
       "      <td>0.050041</td>\n",
       "      <td>0</td>\n",
       "      <td>Cc1c(cc(c2c1oc-3c(c(=O)c(c(c3n2)C(=O)[C@@H]4CC...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>0.951547</td>\n",
       "      <td>0.048453</td>\n",
       "      <td>0</td>\n",
       "      <td>CSc1ncc(c(n1)C(=O)OCc2nc3ccccc3c(n2)N)Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>0.954606</td>\n",
       "      <td>0.045394</td>\n",
       "      <td>0</td>\n",
       "      <td>c1ccc2c(c1)cc(o2)C3=NN([C@@H](C3)c4ccco4)C(=O)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>0.955823</td>\n",
       "      <td>0.044177</td>\n",
       "      <td>0</td>\n",
       "      <td>CCc1nc2c3ccccc3oc2c(n1)N4CCN(CC4)c5ccccc5F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>0.960468</td>\n",
       "      <td>0.039532</td>\n",
       "      <td>0</td>\n",
       "      <td>c1ccc2c(c1)cccc2C(=O)NNc3c(c(ncn3)Oc4cccc5c4cc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>0.960618</td>\n",
       "      <td>0.039381</td>\n",
       "      <td>0</td>\n",
       "      <td>C/C(=N\\N=C(\\c1c(non1)N)/N)/c2ccc(cc2)OCc3cccc4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.967507</td>\n",
       "      <td>0.032493</td>\n",
       "      <td>0</td>\n",
       "      <td>Cn1cc(c2c1cccc2)C(=O)NNC(=O)NCCc3ccccc3Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>0.970460</td>\n",
       "      <td>0.029540</td>\n",
       "      <td>0</td>\n",
       "      <td>COc1ccc(c(c1OC)OC)/C=N/c2c(c3c(s2)CCCC3)C(=O)N...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          neg       pos  active  \\\n",
       "1    0.030795  0.969205       1   \n",
       "3    0.248293  0.751707       1   \n",
       "5    0.424761  0.575239       1   \n",
       "6    0.490254  0.509746       1   \n",
       "2    0.584939  0.415061       1   \n",
       "265  0.643711  0.356289       0   \n",
       "87   0.735412  0.264588       0   \n",
       "0    0.786704  0.213296       1   \n",
       "181  0.892009  0.107991       0   \n",
       "254  0.896371  0.103629       0   \n",
       "271  0.913182  0.086818       0   \n",
       "262  0.918457  0.081543       0   \n",
       "198  0.926560  0.073440       0   \n",
       "4    0.927561  0.072439       1   \n",
       "42   0.947998  0.052002       0   \n",
       "53   0.948888  0.051112       0   \n",
       "269  0.949782  0.050218       0   \n",
       "453  0.949959  0.050041       0   \n",
       "419  0.951547  0.048453       0   \n",
       "161  0.954606  0.045394       0   \n",
       "272  0.955823  0.044177       0   \n",
       "108  0.960468  0.039532       0   \n",
       "379  0.960618  0.039381       0   \n",
       "79   0.967507  0.032493       0   \n",
       "461  0.970460  0.029540       0   \n",
       "\n",
       "                                                SMILES  \n",
       "1    Cc1cnc(nc1c2cc([nH]c2)C(=O)N[C@H](CO)c3cccc(c3...  \n",
       "3    Cn1c-2c(c(n1)C(=O)N)CCc3c2nc(nc3)NC4CCN(CC4)C(...  \n",
       "5    CNC(=O)Nc1ccc(cn1)CNc2c(scn2)C(=O)Nc3ccc4c(c3)...  \n",
       "6      CCN(C)C(=O)c1cc(c[nH]1)c2c(cn[nH]2)c3cccc(c3)Cl  \n",
       "2        Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl  \n",
       "265         CCN[C@H](c1cccs1)c1cc2[nH]c(=O)[nH]c2cc1Br  \n",
       "87    Cc1c2c([nH]c1C(=O)NCCN3c4ccccc4N[C@@H]3C)CCCC2=O  \n",
       "0    Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCC...  \n",
       "181     c1ccc(cc1)c2nc([nH]n2)CN3c4ccccc4N[C@@H]3C5CC5  \n",
       "254                    c1c(c(cc(c1N)F)F)C(=O)N(CCO)CCO  \n",
       "271      c1cc(oc1)C(=O)N(c2nc3ccc(cc3s2)F)/N=C/c4cccs4  \n",
       "262            N=C(N)c1cccc(OCCCOc2cccc(C(F)(F)F)c2)c1  \n",
       "198  N=c1[nH]c(N2CCN3C[C@@H](CNCc4cccnc4)CC[C@H]3C2...  \n",
       "4     Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N  \n",
       "42                 c1c(cc(cc1Cl)Cl)n2c3c(cn2)c(ncn3)NN  \n",
       "53      c1ccc2c(c1)nc(s2)Nc3c(c(ncn3)Nc4cccc5c4nccc5)N  \n",
       "269             Cc1ccccc1c2nc(on2)CN3CCN(CC3)c4ccccc4C  \n",
       "453  Cc1c(cc(c2c1oc-3c(c(=O)c(c(c3n2)C(=O)[C@@H]4CC...  \n",
       "419           CSc1ncc(c(n1)C(=O)OCc2nc3ccccc3c(n2)N)Cl  \n",
       "161  c1ccc2c(c1)cc(o2)C3=NN([C@@H](C3)c4ccco4)C(=O)...  \n",
       "272         CCc1nc2c3ccccc3oc2c(n1)N4CCN(CC4)c5ccccc5F  \n",
       "108  c1ccc2c(c1)cccc2C(=O)NNc3c(c(ncn3)Oc4cccc5c4cc...  \n",
       "379  C/C(=N\\N=C(\\c1c(non1)N)/N)/c2ccc(cc2)OCc3cccc4...  \n",
       "79           Cn1cc(c2c1cccc2)C(=O)NNC(=O)NCCc3ccccc3Cl  \n",
       "461  COc1ccc(c(c1OC)OC)/C=N/c2c(c3c(s2)CCCC3)C(=O)N...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_df.sort_values(\"pos\",ascending=False).head(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fce22f670b8>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEHCAYAAACjh0HiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAARS0lEQVR4nO3df6xfdX3H8eeLWyA4xc5yZ9iFWrT1R+d00Ss/jCJGy4qaNCZuglMQNQ1RSrNkRqLZmCEa0S0RK1oqIcL+EN1GFJeuBBhCnJJRUMDK0LsqcIsZ5WJUkIFt3/vjfquX22/rVXp6bvk8H8lN7+eczz3f121v+rrnx/ecVBWSpHYd0ncASVK/LAJJapxFIEmNswgkqXEWgSQ1ziKQpMYt6GrDSS4H3gI8WFUvHbI+wMXAm4BfAu+uqtt/23aPOuqoWrJkyX5OK0lPb7fddttDVTU6bF1nRQB8EfgscOVe1p8GLBt8nAB8fvDnPi1ZsoTNmzfvp4iS1IYk9+5tXWeHhqrqZuDhfUxZBVxZ024BFiY5uqs8kqTh+jxHMAbcP2M8OVgmSTqA+iyCDFk29H4XSVYn2Zxk8/bt2zuOJUlt6bMIJoFjZ4yPAR4YNrGqNlTVeFWNj44OPdchSfo99VkE1wBnZtqJwM+q6ic95pGkJnV5+eiXgFOAo5JMAhcAhwJU1XpgI9OXjk4wffno2V1lkSTtXWdFUFVn/Jb1BXygq9eXJM1Nl+8jkHQQWrduHRMTE71m2LZtGwBjY/1fSLh06VLWrFnTd4xOWQSS5p3HHnus7whNsQgkPcl8+O137dq1AFx88cU9J2mDN52TpMZZBJLUOItAkhpnEUhS4ywCSWqcRSBJjbMIJKlxFoEkNc4ikKTGWQSS1DiLQJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJapxFIEmNswgkqXEWgSQ1ziKQpMZZBJLUOItAkhpnEUhS4ywCSWpcp0WQZGWSe5JMJDl/yPpnJ/l6kjuSbElydpd5JEl76qwIkowAlwCnAcuBM5IsnzXtA8D3q+rlwCnAPyY5rKtMkqQ9dblHcDwwUVVbq+oJ4Cpg1aw5BTwrSYBnAg8DOzrMJEmapcsiGAPunzGeHCyb6bPAS4AHgLuAtVW1q8NMkqRZuiyCDFlWs8Z/DnwX+GPgz4DPJjlyjw0lq5NsTrJ5+/bt+z+pJDWsyyKYBI6dMT6G6d/8ZzobuLqmTQA/Al48e0NVtaGqxqtqfHR0tLPAktSiLovgVmBZkuMGJ4BPB66ZNec+4A0ASZ4LvAjY2mEmSdIsC7racFXtSHIucC0wAlxeVVuSnDNYvx64EPhikruYPpT0oap6qKtMkqQ9dVYEAFW1Edg4a9n6GZ8/AJzaZQZJ0r75zmJJapxFIEmNswgkqXEWgSQ1ziKQpMZZBJLUOItAkhpnEUhS4ywCSWqcRSBJjbMIJKlxFoEkNc4ikKTGWQSS1DiLQJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJapxFIEmNswgkqXEWgSQ1ziKQpMZZBJLUOItAkhq3oO8AkqatW7eOiYmJvmPMC7v/HtauXdtzkvlh6dKlrFmzprPtd1oESVYCFwMjwGVV9Ykhc04BPg0cCjxUVa/rMpM0X01MTPDDLd9h8TN39h2ld4f9avpgxeP3bu45Sf/ue2Sk89forAiSjACXACuASeDWJNdU1fdnzFkIfA5YWVX3JfmjrvJIB4PFz9zJh1/x875jaB75+O1Hdv4aXZ4jOB6YqKqtVfUEcBWwatacdwBXV9V9AFX1YId5JElDdFkEY8D9M8aTg2UzvRD4wyTfSHJbkjM7zCNJGqLLcwQZsqyGvP4rgTcARwDfTnJLVf3gSRtKVgOrARYvXtxBVElqV5d7BJPAsTPGxwAPDJmzqaoeraqHgJuBl8/eUFVtqKrxqhofHR3tLLAktajLIrgVWJbkuCSHAacD18ya8zXgtUkWJHkGcAJwd4eZJEmzdHZoqKp2JDkXuJbpy0cvr6otSc4ZrF9fVXcn2QTcCexi+hLT73WVSZK0p07fR1BVG4GNs5atnzX+FPCpLnNIkvbOW0xIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJapxFIEmNswgkqXFzKoIkn0xyZJJDk9yQ5KEk7+w6nCSpe3PdIzi1qn4OvIXph8m8EPhgZ6kkSQfMXIvg0MGfbwK+VFUPd5RHknSAzfV5BF9P8t/AY8D7k4wC/9ddLEnSgTKnPYKqOh84CRivql8BjwKrugwmSTow5rRHkORQ4F3AyUkAbgLW7/OLJEkHhbkeGvo80+cJPjcYv2uw7H1dhJIkHThzLYJXVdXLZ4z/I8kdXQSSJB1Yc71qaGeSF+weJHk+sLObSJKkA2muewQfBG5MsnUwXgKc3UkiSdIBNdci+E/gUuANg/GlwLc7SSQ1atu2bTz6ixE+fvuRfUfRPHLvL0b4g23bOn2NuRbBlcDPgQsH4zOAfwL+ootQkqQDZ65F8KJZJ4tv9GSxtH+NjY3x+I6f8OFX/LzvKJpHPn77kRw+Ntbpa8z1ZPF3kpy4e5DkBKYPF0mSDnJz3SM4ATgzyX2D8WLg7iR3AVVVL+sknSSpc3MtgpWdppAk9WZORVBV93YdRJLUD59QJkmNswgkqXEWgSQ1rtMiSLIyyT1JJpKcv495r0qyM8nbuswjSdpTZ0WQZAS4BDgNWA6ckWT5XuZdBFzbVRZJ0t51uUdwPDBRVVur6gngKoY/1WwN8K/Agx1mkSTtRZdFMAbcP2M8OVj2a0nGgLfyW552lmR1ks1JNm/fvn2/B5WklnVZBBmyrGaNPw18qKr2+WyDqtpQVeNVNT46OrrfAkqS5v7O4t/HJHDsjPExwAOz5owDVw2eg3wU8KYkO6rqqx3mkiTN0GUR3AosS3IcsA04HXjHzAlVddzuz5N8Efg3S0CSDqzOiqCqdiQ5l+mrgUaAy6tqS5JzBuv3eV5AknRgdPo+gqraWFUvrKoXVNXHBsvWDyuBqnp3Vf1Ll3n0ZFNTU5x33nlMTU31HUVSj3xnccOuuOIK7rrrLq688sq+o0jqkUXQqKmpKTZt2kRVsWnTJvcKpIZZBI264oor2LVrFwA7d+50r0BqmEXQqOuvv54dO3YAsGPHDq677rqeE0nqi0XQqDe+8Y0sWDB90diCBQtYsWJFz4kk9cUiaNRZZ53FIYdM//OPjIxw5pln9pxIUl8sgkYtWrSIlStXkoSVK1eyaNGiviNJ6kmX7yzWPHfWWWfx4x//2L0BqXEWQcMWLVrEZz7zmb5jSOqZh4YkqXEWgSQ1ziKQpMZZBJLUOItAkhpnEUhS4ywCSWqcRSBJjbMIJKlxFkHDfFSlJLAImuajKiWBRdAsH1UpaTeLoFE+qlLSbhZBo3xUpaTdLIJG+ahKSbtZBI3yUZWSdrMIGuWjKiXt5hPKGuajKiWBRdA0H1UpCTw0JEnNswgkqXEWgSQ1rtMiSLIyyT1JJpKcP2T9XyW5c/DxrSQv7zKPJGlPnRVBkhHgEuA0YDlwRpLls6b9CHhdVb0MuBDY0FUeSdJwXe4RHA9MVNXWqnoCuApYNXNCVX2rqn46GN4CHNNhHknSEF0WwRhw/4zx5GDZ3rwX+PcO80iShujyfQQZsqyGTkxez3QRvGYv61cDqwEWL168v/JJkuh2j2ASOHbG+BjggdmTkrwMuAxYVVVDb4pfVRuqaryqxkdHRzsJK0mt6rIIbgWWJTkuyWHA6cA1MyckWQxcDbyrqn7QYRZJ0l50dmioqnYkORe4FhgBLq+qLUnOGaxfD/wdsAj4XBKAHVU13lUmSdKeOr3XUFVtBDbOWrZ+xufvA97XZQbpYHLfIyN8/PYj+47Ru//95fTBiuc+Y1fPSfp33yMjLOv4NbzpnDRPLF26tO8I88YTExMAHP48/06W0f3PhkUgzRNr1qzpO8K8sXbtWgAuvvjinpO0wXsNSVLjLAJJapxFIEmNswgkqXEWgSQ1ziKQpMZZBJLUOItAkhpnETRsamqK8847j6mpoTd9ldQIi6Bhl156KXfeeScbNviEUKllFkGjpqamuP766wG47rrr3CuQGmYRNOrSSy9l167pOzvu2rXLvQKpYRZBo2644YYnjXfvHUhqj0XQqMGDgPY6ltQOi6BRr3nNa/Y5ltQOi0CSGmcRNOqb3/zmPseS2mERNKqq9jmW1A6LoFFHH330PseS2mERNGr2G8h8Q5nULougUStWrHjS+NRTT+0piaS+WQSNOvnkk/c5ltQOi6BRn/zkJ580vuiii3pKIqlvFkGjHnzwwX2OJbXDIpCkxlkEktQ4i0CSGreg7wCS5pd169YxMTHRa4bdr7927dpecwAsXbqUNWvW9B2jUxaBpHnniCOO6DtCUywCSU/ydP/tV3vq9BxBkpVJ7kkykeT8IeuT5DOD9XcmeUWXeSRJe+qsCJKMAJcApwHLgTOSLJ817TRg2eBjNfD5rvJIkobr8tDQ8cBEVW0FSHIVsAr4/ow5q4Ara/oeyLckWZjk6Kr6SVeh5sOJsG3btvHYY4/1mmGYN7/5zb287hFHHMHY2Fgvrz1TCycFpWG6LIIx4P4Z40nghDnMGQM6K4KbbrqJhx56qKvNH9QeffTR3l53PvybbNu2zSJQk7osgmFPQ5/99JO5zCHJaqYPHbF48eKnFGrhwoW9/zb++OOPs2vXrl4zDHv9Qw7p520lhxxyCIcffngvrz3TwoUL+44g9aLLIpgEjp0xPgZ44PeYQ1VtADYAjI+PP6VHaV122WVP5cufNj760Y9y4403/nq8YsUKPvKRj/SYSFJfuvwV8FZgWZLjkhwGnA5cM2vONcCZg6uHTgR+1uX5Af3GBRdc8KSxJSC1q7MiqKodwLnAtcDdwFeqakuSc5KcM5i2EdgKTABfAN7fVR7t6fWvfz2w50NqJLUlB9tDy8fHx2vz5s19x5Ckg0qS26pqfNg6bzonSY2zCCSpcRaBJDXOIpCkxlkEktS4g+6qoSTbgXv7zvE0chTQ//0dpD35s7l/Pa+qRoetOOiKQPtXks17u6RM6pM/mweOh4YkqXEWgSQ1ziLQhr4DSHvhz+YB4jkCSWqcewSS1DiLoFFJVia5J8lEkvP7ziPtluTyJA8m+V7fWVphETQoyQhwCXAasBw4I8nyflNJv/ZFYGXfIVpiEbTpeGCiqrZW1RPAVcCqnjNJAFTVzcDDfedoiUXQpjHg/hnjycEySQ2yCNqUIcu8fExqlEXQpkng2BnjY4AHesoiqWcWQZtuBZYlOS7JYcDpwDU9Z5LUE4ugQVW1AzgXuBa4G/hKVW3pN5U0LcmXgG8DL0oymeS9fWd6uvOdxZLUOPcIJKlxFoEkNc4ikKTGWQSS1DiLQJIaZxFIT0GSU5K8esb4nCRn9plJ+l0t6DuAdJA7BXgE+BZAVa3vNY30e3CPQBoiyVeT3JZkS5LVg2Urk9ye5I4kNyRZApwD/HWS7yZ5bZK/T/I3SV6S5L9mbG9JkjsHn78yyU2D7V+b5Og+vkdpN/cIpOHeU1UPJzkCuDXJ14AvACdX1Y+SPGewfj3wSFX9A0CSNwBU1d1JDkvy/KraCrwd+EqSQ4F1wKqq2p7k7cDHgPf08U1KYBFIe3NekrcOPj8WWA3cXFU/Aqiqudwv/yvAXwKfYLoI3g68CHgpcF0SgBHgJ/s3uvS7sQikWZKcArwROKmqfpnkG8AdTP8n/rv4MvDPSa4Gqqp+mORPgS1VddL+zCw9FZ4jkPb0bOCngxJ4MXAicDjwuiTHASR5zmDuL4BnDdtIVf0PsBP4W6ZLAeAeYDTJSYPtHJrkTzr7TqQ5sAikPW0CFgxO7l4I3AJsZ/rw0NVJ7uA3/7F/HXjr7pPFQ7b1ZeCdTB8mYvBo0LcBFw22813g1UO+TjpgvPuoJDXOPQJJapxFIEmNswgkqXEWgSQ1ziKQpMZZBJLUOItAkhpnEUhS4/4fDXuGgsdVMYoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(pred_df.active,pred_df.pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance of our model is very good, we can see a clear separation between the active and inactive compounds.  It appears that only one of our active compounds receieved a low positive score. Let's look more closely. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "false_negative_df = pred_df.query(\"active == 1 & pos < 0.5\").copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "PandasTools.AddMoleculeColumnToFrame(false_negative_df,\"SMILES\",\"Mol\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg</th>\n",
       "      <th>pos</th>\n",
       "      <th>active</th>\n",
       "      <th>SMILES</th>\n",
       "      <th>Mol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.786704</td>\n",
       "      <td>0.213296</td>\n",
       "      <td>1</td>\n",
       "      <td>Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCCN5CCOCC5)OC</td>\n",
       "      <td><img src=\"\" alt=\"Mol\"/></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.584939</td>\n",
       "      <td>0.415061</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl</td>\n",
       "      <td><img src=\"\" alt=\"Mol\"/></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.927561</td>\n",
       "      <td>0.072439</td>\n",
       "      <td>1</td>\n",
       "      <td>Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N</td>\n",
       "      <td><img src=\"\" alt=\"Mol\"/></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.971956</td>\n",
       "      <td>0.028044</td>\n",
       "      <td>1</td>\n",
       "      <td>c1ccc2c(c1)c(c[nH]2)C[C@@H](COc3cc(cnc3)c4ccc5cnccc5c4)N</td>\n",
       "      <td><img src=\"\" alt=\"Mol\"/></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        neg       pos  active  \\\n",
       "0  0.786704  0.213296       1   \n",
       "2  0.584939  0.415061       1   \n",
       "4  0.927561  0.072439       1   \n",
       "7  0.971956  0.028044       1   \n",
       "\n",
       "                                              SMILES  \\\n",
       "0  Cn1ccnc1Sc2ccc(cc2Cl)Nc3c4cc(c(cc4ncc3C#N)OCCC...   \n",
       "2      Cc1c2cc(c(cc2oc(=O)c1Cc3ccccc3)OC(=O)N(C)C)Cl   \n",
       "4   Cc1c2cc(ccc2[nH]n1)c3cc(cnc3)OC[C@H](Cc4ccccc4)N   \n",
       "7  c1ccc2c(c1)c(c[nH]2)C[C@@H](COc3cc(cnc3)c4ccc5...   \n",
       "\n",
       "                                                 Mol  \n",
       "0  <img src=\"...  \n",
       "2  <img src=\"...  \n",
       "4  <img src=\"...  \n",
       "7  <img src=\"...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "false_negative_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "false_positive_df = pred_df.query(\"active == 0 & pos > 0.5\").copy()\n",
    "PandasTools.AddMoleculeColumnToFrame(false_positive_df,\"SMILES\",\"Mol\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg</th>\n",
       "      <th>pos</th>\n",
       "      <th>active</th>\n",
       "      <th>SMILES</th>\n",
       "      <th>Mol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [neg, pos, active, SMILES, Mol]\n",
       "Index: []"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "false_positive_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've evaluated our model's performance we can retrain the model on the entire dataset and save it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6702756252973592"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "nteract": {
   "version": "nteract-on-jupyter@1.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
